Method and apparatus for image-based eye tracking for retinal diagnostic or surgery device

ABSTRACT

An eye tracking method for determining a position of an eye or a part of an eye in an image of an image sequence by performing a comparison between said image and a reference image, said process including:
         aligning a set of images;   computing an enhanced reference image based on a combination of said set of aligned images; and   determining said position in said image of said image sequence by comparing said image of said image sequence and said enhanced reference image to yield a motion estimation between said reference image and said image of said sequence.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. patent application Ser. No.11/142,962, filed Jun. 1, 2005, entitled METHOD AND APPARATUS FORIMAGE-BASED EYE TRACKING FOR RETINAL DIAGNOSTIC OR SURGERY DEVICE, whichclaims benefit of European Patent application 04013015.5, filed Jun. 2,2004, which are incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to a method and an apparatus used inimage-based tracking of eye movement or position in a retinaldiagnostic/surgical device.

BACKGROUND OF THE INVENTION

Image-based tracking of eye movement or position is known to be used forseveral applications such as surgery and diagnostics, one example beinga retinal diagnostic/surgical device. However, conventional eye trackingtechniques suffer from several problems. One problem is the imagequality provided by the system, e.g. the signal-to-noise ratio SNR. Ifthe SNR is low, then the image processing techniques used for eyetracking are not as accurate as they could be. Moreover, there areproblems which are due to the specific instrumentational configuration.

One such example is retinal video tracking for a Optical CoherenceTomography (OCT) device for measuring thickness and structure of theretinal layers, but other specific fields such as devices for surgery,diagnosis or monitoring of the eye can be mentioned here as well.

In case of an OCT system, the measurement beam of the OCT interfereswith the fundus imaging due to fact that the illumination spectrum ofthe beam is close to the imaging spectrum (near IR). The effects of theOCT beam can be described as having two components:

-   -   Saturations of the image on small areas where the beam hits the        retinal surface    -   Low frequency illumination changes that are caused by beam light        diffusion at the retinal surface and beam light reflection from        retinal surface.

Additionally image is corrupted by acquisition noise and by illuminationdistortions caused by the pupil size and alignment changes during theprocedure. can be applied to any device in which either a measurementscanning beam or surgical laser beam effects the retinal image used fortracking or alignment.

It is therefore an object of the present invention to provide a methodand apparatus which is capable to enhance the performance of aconventional eye tracking system.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention there is providedan eye tracking method for determining a position of an eye or an eyefeature such as the retina in an image of an image sequence byperforming a comparison between said image and a reference image, saidprocess including:

-   -   aligning a set of images;    -   computing an enhanced reference image based on a combination of        said set of aligned images; and    -   determining said position in said image of said image sequence        by comparing said image of said image sequence and said enhanced        reference image to yield a motion estimation between said        reference image and said image of said sequence.

By using an enhanced reference image it is possible to improve theoverall accuracy of the eye tracking method, thereby enhancing itsperformance.

In case of an OCT system thereby it becomes possible to allow trackingdespite the use of disturbed (real-time) images without necessitatingany hardware changes to the optics or hardware to reduce the influenceof the scanning beam due to the use of an enhanced reference image forcomparison with the real-time images.

A set of images taken in advance may be aligned to the same retinalposition and then combined by averaging the aligned set of images togenerate the enhanced reference image. This enhanced reference image maythen be used for position determination and tracking in combination witha diagnosis and/or surgical device The diagnosis and/or surgical devicein one embodiment may be one of the following:

-   -   an OCT apparatus;    -   a refractive surgery device.

According to one embodiment there is performed an OCT spot tracerectification to eliminate a saturated OCT spot trace from an image insaid sequence by replacing the saturated area by an interpolation or anaverage intensity in a corresponding area of unsaturated imagescompensating illumination variations using one or more suitable filters.This reduces the disturbing effect of the OCT spot.

According to an embodiment a motion estimation comprises:

performing a global motion estimation between said reference image andsaid image of said sequence; and

performing a motion estimation based on parts or landmarks of saidreference image and said image of said image sequence. This two-stepmechanism yields a better accuracy and is less prone to disturbingeffects.

In one embodiment said global motion estimation is refined by saidmotion estimation based on parts or landmarks. They may be selected bythe user or they may be automatically selected.

In one embodiment the enhanced reference image is used for one or moreof the following:

OCT spot rectification;

deriving a spectrally matched filter for selecting one or more retinaspectrum components having the highest SNR ratio.

In one embodiment the method further comprises:

aligning image sequences taken at different periods of time by aligningan enhanced reference corresponding to one of said sequences with areference image or an enhanced reference image corresponding to saidsecond sequence. This enables an inter-session alignment necessary for along-term diagnosis.

According to one embodiment there is provided a method of enhancing thevideo display of an image of an eye or a portion of an eye, said methodcomprising:

aligning a sequence of images which are preceding an actual image ofsaid video stream with respect to said actual image;

combining the resulting aligned set of images into an enhanced image tobe used as a basis for an enhanced actual image of said video stream.Thereby the quality of the video stream may be enhanced.

According to one embodiment the method comprises aligning said images ofsaid preceding sequence based on eye tracking information, and for theeye tracking the enhanced reference image may be used.

According to one embodiment there is provided a method for operating adiagnostic or surgical device based on eye tracking, said methodcomprising:

determining whether an anomaly is detected during said eye tracking;

if an anomaly is detected during said eye tracking, putting saiddiagnostic or surgical device in a HOLD mode. This enables the system toavoid the negative influence of anomalies in eye motion such as saccadesor links.

According to one embodiment an anomaly is one or more of the following:

-   -   a change in image quality beyond a certain threshold;    -   a fast movement of the eye the speed of which is beyond a        certain threshold;    -   a blurring of the image used for eye tracking;    -   a blurring of the image used for eye tracking in combination        with a fast movement;    -   a cumulated difference between subsequent tracking images lying        beyond a certain threshold.

In one embodiment the method may further comprise:

determining those locations where the operation of the surgical ordiagnostic device did nod lead to a correct result;

re-running the operation of said surgical or diagnostic device for saiddetermined locations.

Further embodiments comprise systems and apparatuses which implement themethods according to embodiments of the invention.

DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates a configuration for implementing anembodiment of the present invention.

FIG. 2 schematically illustrates a method according to an embodiment ofthe invention.

FIG. 3 schematically illustrates a method according to a furtherembodiment of the invention.

FIG. 4 schematically illustrates a motion estimation method according toa further embodiment of the invention.

FIG. 5 schematically illustrates a method according to a furtherembodiment of the invention.

DETAILED DESCRIPTION

A first embodiment of the present invention will now be described inconnection with FIG. 1. A camera (e.g. a CCD camera) 110 takes videoimages of an eye 100, and the images are fed to a computer 120. Such anarrangement is equal to a classical eye tracking system and it may alsobe applied to the present invention. In the conventional eye trackingsystem, based on the images captured the computer 120 “tracks” the eyemovement, e.g. by comparing the position of some “landmarks” in themomentary image with the position in the previous image to thereby“track” the movement of the eye. Based on the tracked actual position afeedback signal may be determined which then is used to carry out aposition compensation at a surgical or diagnostic device (not shown inFIG. 1).

In order to track the movement of the eye or to determine the positionof the eye an instantaneous image of the eye is compared with somereference image. According the present embodiment this reference imageis derived by combining a plurality of images into an enhanced referenceimage. This is now explained in somewhat more detail in connection withFIG. 2.

In operation 200 a plurality of “initial images” is taken, e.g. bycapturing an image sequence before the actual tracking starts. E.g. for0.5 seconds before the actual tracking the camera captures an imagesequence, this yields a set of “initial images” based on which theenhanced reference image then will be derived.

In operation 210 the initial images then are aligned with respect totheir position. This can be done using some standard motion estimationmethod for each of the initial images and then aligning them based onthe result.

In operation 220 the thus aligned initial images then are combined intoan enhanced reference image, e.g. by averaging the initial images into asingle image which then has an enhanced SNR.

In operation 230 the enhanced reference image then is used for trackingor position measurement, e.g. by comparing the position of aninstantaneous image with the enhanced reference image.

According to one embodiment the eye tracking consists of two majorelements which are as follows:

-   -   A. Image set alignment    -   B. Image sequence alignment

The image set alignment takes a set of (e.g.) retinal images as inputand produces an enhanced (retinal) image as output by combining theinformation in the image set. The image set alignment in one embodimentis done before the actual tracking and does not bare real-timeconstraints so that the method used to implement in a manner oriented tomaximize the quality of the result.

The image sequence alignment makes use of the enhanced retinal imageformerly produced, in order to compare a sequence of new retinal imagesto the mentioned enhanced retinal image for determination of eyeposition/eye movement of to perform eye tracking. The image sequencealignment has real-time constraints (due to the continuous input ofreal-time instantaneous images) and according to one embodiment isoriented for an optimal compromise between motion determination qualityand computational effort.

In the following another embodiment will be described in somewhat moredetail which relates to an application in the field of optical coherencetomography (OCT). An OCT system works similar to an ultrasound imagingdevice except that instead of ultrasound it uses infrared (IR) lightwhich is emitted as OCT beam, reflected by the retina and then based onthe reflected light and image processing methods a diagnostic image isobtained. Such systems are well known to a person skilled in the art andare therefore not described any further herein.

In both set and sequence alignment in an OCT system a main problem isthat of the adverse effects of OCT beam. A preprocessing method thatcompensates for these effects is described in the following. Accordingto one embodiment this method comprises the acquisition of the image setused for deriving the enhanced reference image, furthermore it comprisesan illumination normalization and an OCT spot trace rectification.

At first the preprocessing method will be described. Later then theimage set/sequence alignment will be described.

The preprocessing method in one embodiment consists of followingsub-steps:

-   -   1. Select a small set (or subset) of images at the beginning of        the procedure    -   2. Preprocess each image in order to reduce the illumination        distortion effects    -   3. Preprocess each image to rectify the OCT spot trace        (saturated pixels)    -   4. Align the image set with respect to each other and average        the set temporary to reduce the effect of acquisition noise as        well as other residual distortions

The subset of images is acquired previous to the beginning of the scanmode—in the scan alignment mode. In one embodiment approx. 0.5 secondsof live video stream are taken that are buffered into memory andprocessed afterwards. If images during this period are suffering frombad quality (very low contrast or large saturated areas) they arerejected.

Then the preprocessing proceeds with an illumination normalization. Theillumination normalization in one embodiment is performed as acombination of homomorphic and high pass filtering. Namely, the signalpixels are altered by a multiplicative or/and additive influence of theillumination strength. If light is reflected from retina, the effect ismultiplicative (i.e. contrast increases). If light is diffused by retinafrom punctual sources, the effect is additive (i.e. brightnessincreases).

In practice both effects appear. In the pupil vignetted areas the effectis mainly multiplicative, while close to the beam trace the effect ismore additive. An illumination normalization attempts to reduce theseeffects, and in one embodiment the method can operate as follows:

-   -   1. Perform a low pass filtering with a large kernel (e.g. 10-20        pixels). The result represents the illumination variation Ilum.        Standard methods known to the expert can be used to perform the        low pass filtering.    -   2. Correct each pixels after the relation:

Im(x,y)=alpha*(Im(x,y)/Ilum(x,y))+(1−alpha)*(Im(x,y)−Ilum(x,y))

-   -   The parameter alpha controls the weight on        multiplicative/additive correction and varies with the position        in the image. More precisely the alpha value is depending on the        intensity of pixels in Im(x,y) according to a predefined        function. Typically for dark pixels it will chosen to be closer        to 1, while for pixels close to the saturation it will be chosen        to be closer to 0. Both additive and multiplicative image        correction methods are well known in the art. The combination of        the two models is also straightforward to the skilled person.        Details can be found in any image processing textbook, e.g:        Anil K. Jain, Fundamentals of Digital Image Processing, Prentice        Hall 1988. (Note: for simplicity, the given equation omits        certain fixed offset/scaling factors used for making the        multiplicative/additive images comparable)

It should be noted that the low pass filter on the above will causeringing in the vicinity of the OCT beam spot, because the illuminationvaries very abruptly in those areas.

The treatment of saturated pixels requires therefore a differentcompensation (or “correction”) approach. In one embodiment thesaturation effects can be dealt with using a method which comprises thefollowing steps:

-   -   1. The OCT spot trace is first identified by performing an image        thresholding to detect the very bright pixels.    -   2. The binary image is blurred with a kernel identical to the        one used at the illumination normalization step. All pixels that        are non zero in this mask image, represent non-valid pixels        (corresponding to saturated pixels). The blurring takes into        account that the thresholding may “overlook” some pixels at the        edge of saturated areas which in fact should also be considered        as needing compensation. For the blurring any standard “blurring        method” known to experts in the field of image processing may be        used.    -   3. The entire set of illumination normalized images is averaged        over time, taking care that non-valid pixels are ignored. The        non-valid pixels are treated differently as described in the        following.    -   4. Each image is rectified in the non-valid pixels as an        alpha-blending between the image and the average image. The        alpha factor is represented by the mask image computed during        the illumination normalization process.

After illumination normalization the preprocessing according to oneembodiment is completed and the method may now proceed with aligning theinitial set or subset of images which has been preprocessed.

Direct image to image alignment is difficult in low SNR images.According to an embodiment of the invention therefore an enhancedreference image is derived to which then is used for eye positiondetermination. This improves SNR and yields better accuracy.

In detail, according to one embodiment alignment of the initial set forobtaining the enhanced reference image in order to improve SNR operatesas follows. At first one can perform a large set of measurements thatare redundant and proceed by solving an over determined system ofequations. Given n images, such approach would require to compute thedisplacement between each pair of images (1, 2) (1,3) . . . (1,n),(2,3), (2,4) . . . (2,n) . . . (n−1,n). This leads to n(n+1)/2 imagepairs comparisons to obtain the displacements of the individual imageswhich then may be aligned and added or averaged. However, in terms oftime necessary to perform the calculation this is a kind of “bruteforce” method which in principle works but which can—according tofurther embodiments—be replaced by more efficient calculation methods.

For example, it can be noticed that in case of linear operators (such ascorrelation family), the order of operations can be inversed, namely onecan first add k images and than compute the displacement, instead ofcomputing k displacements and then adding results. This approach worksas long as the k images are not moving much with respect to each other.

However, eye movement research shows the following facts:

-   -   motion during short fixations are small    -   fixations durations is of at least 100 ms    -   saccades/flicks during fixations are less than 30 ms    -   saccades/flicks during fixations are low frequency<3/sec

Based on this according to one embodiment can proceed as follows:

-   -   1. Divide an initial set of 32 images in 4 subsets of 8 images.        Each subset represents a time span of 128 ms.    -   2. Average the each subset to obtain an average image per        subset.    -   3. Measure the displacement of an image with respect to the 3        average images from the subsets that it does not belong.    -   4. Take the median of the 3 measurements as the actual        displacement    -   5. Based on the results obtained shift each image so that the        set is now aligned, and then average all 32 images to obtain the        final reference enhanced image.

The particular numbers in the foregoing example are to be understood asbeing just informative examples, the method can be applied with anynumber of images/subsets, etc.

After having obtained the enhanced reference image as described beforethis enhanced reference image can be used for determining eye positionor eye movement or eye tracking. An example of the use of the enhancedreference image could be in connection with a surgical/diagnostic devicewhich needs a feedback for compensating the movement of the eye,typically in real-time. The operation of such a system is schematicallyillustrated by the flowchart shown in FIG. 3.

In operation 300 a real-time image of the eye (or a part of it, or someof its parts) is obtained, e.g. by use of a system schematically shownin FIG. 1. The thus obtained image is then compared in operation 310with the enhanced reference image, e.g. by using any standard method todetermine the motion or displacement between the two images. Suchmethods for motion estimation are well known to the expert and aretherefore here not described in any more detail. In operation 320 thenthe actual displacement is determined by using the motion estimationmethod applied in operation 310. This then yields the actual position(or the displacement with respect to the reference image) of the imagetaken in operation 300. Based on the displacement in operation 330 thena compensation feedback value is determined, this may e.g. be a signalto be used for controlling the surgical/diagnostic device or itscomponents such that the eye movement is compensated. In operation 340then based on this feedback the actual compensation is performed in thesurgical/diagnostic device. Thereafter the procedure may continue byacquiring the next image by returning to operation 300.

Image sequence alignment (or position measurement) differs from imageset alignment by the fact that the image data is presented in a temporalsequence rather than simultaneously. That means that aligning an imagecan be done only with the information gathered up to the point ofmeasurement. Sequence alignment is the basis of any real-time videotracking.

The sequence alignment method of the present embodiment makes use of thereference enhanced image obtained as described before. According to aparticular embodiment the enhanced reference image is used in sequencealignment or real-time position determination for one or more of thefollowing purposes:

-   -   1. For OCT spot rectification. Similar to the OCT spot        rectification method described before in connection with the        alignment of the initial set of images the OCT rectification can        also be performed for the real-time images obtained during an        OCT scan. The saturated pixels then are replaced by pixel values        which are based on the values of the enhanced reference image        for these pixels.    -   2. For deriving a spectrally matched filter that selects the        retinal spectrum components with the highest SNR ratio. Based on        the enhanced reference image an analysis can be made which        spectral components have the highest SNR. These spectral        components then may also be used for the actual sequence        alignment during real-time data acquisition.    -   3. For detection of useful landmarks such as retinal vessels,        optical nerve, macular exudates. The thus detected landmarks        which have been detected in the enhanced reference image either        manually or automatically may then be used for position        determination during sequence alignment.    -   4. For performing matching techniques (such as phase        correlation, cross correlation, entropy maximization, etc)        between reference image and the currently measured image. These        methods then actually yield the displacement which then may be        used to generate a compensation feedback at the diagnostic or        surgical device. or The matching techniques could be global at        the level of entire image or local, at the level of landmarks as        detected at step 3.

In all mentioned points, the higher quality of the enhanced referenceimage is increasing performance over the use of any particular image ofthe sequence.

A specific embodiment for determining the position of the eye is nowdescribed in the following. Basically the method comprises two majorelements, a first being a motion estimation at a global level, and theresult of this global motion estimation being refined by using a localmotion estimation by using one or more local landmarks detected in thereference image. This is schematically illustrated in FIG. 4, whereoperation 400 shows a global motion estimation based on the whole actualimage and the whole reference image. The resulting estimate suffers frominaccuracies which may result from multiple motions or fromocclusion/saturation effects due to the OCT beam. This estimate is thenin operation 410 refined by the local motion estimation which may bebased on one or more landmarks which have been selected in the enhancedreference image.

In somewhat more detail according to a specific embodiment this may bedescribed as follows.

-   -   1. a global level estimation is performed using the entire        information available. This estimation is performed by matched        phase correlation. Matched phase correlation is a standard        method for motion estimation and well known to the skilled        person. It is yields a set of peaks at positions corresponding        to possible displacements between the actual and the reference        image. The global estimation provides a small set of possible        locations of the retina (1-5 locations) corresponding to        respective possible eye movements. The global step provides        robustness over illumination changes, noise, etc.    -   2. Then there is performed a local level estimation using a set        of landmarks detected in the reference image. This helps to        decide which of the location candidates delivered by step 1 is        the correct one. The local estimation provides flexibility to        adjust over objects occlusions and multiple motions that global        step cannot resolve.

In the following there will be described a further embodiment of thepresent invention which relates to an OCT device providing an OCT imageof the retina. This device makes use of a retina tracker which providesreal-time information about the retina x/y displacements relative to thereference image. The OCT scanning system adjusts the scan pattern totake into account the changes in retina position, so that the scanningpath follows the intended trajectory.

However, standard video based eye trackers (<=60 Hz) are too slow tocorrectly compensate the fast movements of the retina that are typicalto saccades. Also, retinal tracking can be temporary unavailable due toblinks, large pupil occlusions, etc. In order to take into account andto compensate those effects the following two methods of operation areperformed in this embodiment.

-   -   I. The diagnostic/surgery device, and in one particular        embodiment the OCT scan beam is put on hold if an anomaly is        detected in the image sequence used for tracking. The retinal        tracking system detects anomalies in tracking such as saccades,        blinks, etc. (see operation 500 in FIG. 5). This can be done        using certain suitable criteria. E.g. a sudden loss in image        quality (a decrease of a quality factor below a certain        threshold) may be representative of a blink. On the other hand,        a blurring of the image in connection with a fast movement can        be considered as being representative of a saccade. Using such        kind of criteria it can be determined whether an anomaly        occurred which may negatively influence the performance of the        tracking and/or the performance of the surgical/diagnostic        device. In our specific embodiment the device is an OCT device,        and if an anomaly is detected the retinal tracking system in        operation 510 delivers a HOLD signal to the OCT scanning system.        When the HOLD signal is active (as long as operation 520 in FIG.        5 does not lead to an affirmative answer), the OCT scanning        system stops the scanning pattern program. When in operation 520        it is determined that the anomaly is not present anymore, then        the HOLD signal becomes inactive and the scan continues. In this        way, the transient parts are avoided as long as they are        detected by the retinal tracking system.    -   II. A further operation method may be employed additionally or        alternatively. In the present embodiment it is employed        additionally, therefore the system maintains all parts described        at point I. Additionally, based on further data obtained by the        retinal tracking system, such as the future trajectory of the        retina and potentially by the OCT scan data, certain scan parts        can be retrospectively considered invalid. The invalidation is        meant to occur mainly in two situations:        -   a. A detection of a saccade will allow eliminating the            samples from the saccade start, which could not be            eliminated by the first method. The saccades are detected by            means of retinal velocity, as a sequence of 3-5 high-speed            samples. By fitting a predefined parameterized function            (such as a sigmoid or polynomial) to the data samples, a            more precise start/end of the saccade can be obtained        -   b. A detection of accidental tracking errors, that appears            as a disrupter in the trajectory of the retina. These            outlier measurements are detected by means of statistic            filtering. Principally, this comes to evaluating the            probability that a certain position value occurs in a given            trajectory context. The probabilities are computed off line            based on a large dataset of common eye dynamics. The            measurements with low probability are rejected    -   The OCT scanning system then reprograms the reacquisition of        those invalid parts on a second pass. Third pass and so on may        be employed until the desired result is achieved. This method,        may provide more robustness as the decision on where the        scanning was inadequate is based on more data.

In the following a further embodiment of the present invention will bedescribed where the invention is applied in an OCT device for retinalimaging. The diagnosis of a retinal disease usually requires monitoringchanges in the retina over a certain period of time. This implies thatmultiple OCT scans are required to be performed at various intervals onthe same patient in the same location of the retina.

In order to reliably monitor significant changes, it is of highimportance to be able to ensure that different scan sessions areperformed on same retinal location. The current inter-sessions alignmentprocedure is very approximate as the low quality of IR fundus video doesnot allow the doctor to make precise adjustments of the scan placement.The red-free image snapshot capability cannot be used before thediagnosis—only after the diagnosis—as the bright green light constrictsthe pupil. It is, however, desirable to have the OCT system properlyaligned before the scan, and for that the red-free image cannot be usedbecause of its effects on the pupil which would disturb the subsequentdiagnostic measurement.

The enhanced reference image, however, provides enough visual quality tobe used either in a manual or automatic way in order to perform thealignment of the scan with respect to another surgery.

According to an embodiment the system is able to perform the followingautomatic alignments:

-   -   1. Enhanced IR to Enhanced IR    -   2. Enhanced IR to Red Free

The same types of alignment can be performed manually by the doctor whois provided with a software tool that allows free adjustments ofEnhanced IR image in order to highlight various features (retinalvessels, optical nerve head, exudates, etc.). In case of automaticalignment some motion estimation algorithm is used and based on itsresult the alignment is then performed.

The use of the Enhanced Reference Image leads to several benefits:

-   -   It can be used prior to the start of scanning as opposed to the        Red-Free image    -   It offers the required visual quality that allows the operator        to validate/perform the alignment    -   It allows robust automatic alignment with enhanced IR images or        Red-Free images from other sessions

In the following a further embodiment of the invention will be describedwhich relates to the enhancement of the quality of a video image stream.The raw video stream is having a poor SNR that makes typical correctionssuch as brightness/contrast adjustment ineffective.

The present embodiment makes use of the tracking information in order toincrease the SNR of each video frame. Subsequently, other enhancementtechniques (brightness/contrast adjustment, sharpening, shadowcorrection, etc.) may be applied to provide a real time enhanceddisplay.

The method consists of following steps:

For each image in the video stream

-   -   1. Take the last N images (N=10-25) and align them with respect        to the current image, based on the tracking information. The        alignment is performed here by shifting images in x and y        coordinates. Other type of alignment, such as rotation and        scaling or combinations of those can also be employed if        required.    -   2. Replace each pixel value in the display image by the average        (or weighted average) of pixel values in the last N aligned        images. The above two steps can be expressed by the relation:

Displ^(n)(x, y)=Im ^(n)(x, y)+Im ^(n−1)(x−dx ^(n−1) , y−dy ^(n−1))+ . .. Im ^(n−N)(x−dx ^(n−N) , y−dy ^(n−N)),

-   -    where n represents the n-th image of the video stream,        dx^(i)/dy^(i) represents the shift in x/y of the image i        relative to the image n    -   3. Apply the desired enhancement techniques to image Displ and        send it to the display output

With the method described before the quality of each individual videoimage and thereby of the whole video stream can be increased.

The skilled person will understand that the methods, apparatuses andsystems according to embodiments of the invention as describedhereinbefore may be implemented by a configuration comprising a standardvideo camera, and a standard computer as schematically illustrated inFIG. 1. The computer may be equipped with some standard software forvideo capturing, and as far as the foregoing description and the claimsrelate to modules or components implementing the invention the skilledperson will readily understand that they may be implemented either inhardware or in software in connection with the basic configuration shownin FIG. 1. Apart from video eye tracking systems the invention may beapplied to any eye tracking system that outputs eye position data. Basedon the foregoing description the skilled person will be readily able toadapt the system shown in FIG. 1 by suitable programming of the computerand its components to perform the functions described in connection withthe embodiments of the present invention.

It is further to be understood that the foregoing embodiments aredescribed as exemplary embodiments only, and that modifications to theseembodiments are possible to the skilled person and should therefore beconsidered as lying within the scope of the invention. E.g. in additionto an OCT device the invention may be applied to any surgical ordiagnostic device. Moreover, apart from the area of surgical anddiagnostic devices the present invention may be applied in the field ofeye tracking and in eye tracking devices in general.

What is claimed is:
 1. A method of enhancing the video display of animage of an eye or a portion of an eye, said method comprising: aligninga sequence of images which are preceding an actual image of said videostream with respect to said actual image; combining the resultingaligned set of images into an enhanced image to be used as a basis foran enhanced actual image of said video stream.
 2. The method of claim 1,wherein for each actual image of said video stream the display is basedon said enhanced image resulting of a combination of aligned imagespreceding said actual image.
 3. The method of claim 1, furthercomprising: aligning said images of said preceding sequence based on eyetracking information.
 4. The method of claim 3, wherein said trackinginformation is obtained based on an enhanced reference image as definedin claim
 1. 5. A method for operating a diagnostic or surgical devicebased on eye tracking, said method comprising: determining whether ananomaly is detected during said eye tracking; if an anomaly is detectedduring said eye tracking, putting said diagnostic or surgical device ina HOLD mode and/or re-performing the operation of said diagnostic orsurgical device for those parts of the eye where an anomaly has beendetected.
 6. The method of claim 5, wherein said anomaly is one of thefollowing: a change in image quality beyond a certain threshold; a fastmovement of the eye the speed of which is beyond a certain threshold; ablurring of the image used for eye tracking; a blurring of the imageused for eye tracking in combination with a fast movement; a cumulateddifference between subsequent tracking images lying beyond a certainthreshold; a tracking error which is detected based on one or moreimages being considered statistically as outliers; a sequence of imageswith a highs speed retinal movement; an image sequence patternrepresenting a saccade; an image sequence pattern representing a blink.7. The method of claim 5, wherein said determination whether an anomalyis detected is performed in realtime, and if an anomaly is detected saiddiagnostic or surgical device is put into a HOLD mode.
 8. The method ofclaim 5, wherein said anomaly during eye tracking is based onretrospectively evaluating images which have been obtained during eyetracking to identify anomalies, said method further comprising:reperforming the operation of said surgical or diagnostic device onthose locations for which anomalies have been detected.
 9. The method ofclaim 8, wherein said retrospective evaluation comprises: comparingimage sequences obtained during operation of said surgical or diagnosticdevice with eye movement patterns representing common eye movements todetect said anomalies.
 10. An apparatus for enhancing the video displayof an image of an eye or a portion of an eye, said apparatus comprising:a module for aligning a sequence of images which are preceding an actualimage of said video stream with respect to said actual image; a modulefor combining the resulting aligned set of images into an enhanced imageto be used as a basis for an enhanced actual image of said video stream.11. The apparatus of claim 10, wherein for each actual image of saidvideo stream the display is based on said enhanced image resulting of acombination of aligned images preceding said actual image.
 12. Theapparatus of claim 10, further comprising: a module for aligning saidimages of said preceding sequence based on eye tracking information. 13.The apparatus of claim 12, wherein said tracking information is obtainedbased on an enhanced reference image as defined in claim
 20. 14. Anapparatus to be used in connection with a diagnostic or surgical devicewhich operates based on eye tracking, said apparatus comprising: amodule for determining whether an anomaly is detected during said eyetracking; and a module for if an anomaly is detected during said eyetracking, putting said diagnostic or surgical device in a HOLD modeand/or a module for re-performing the operation of said diagnostic orsurgical device for those parts of the eye where an anomaly has beendetected.
 15. The apparatus of claim 14, wherein said anomaly is one ormore of the following: a change in image quality beyond a certainthreshold; a fast movement of the eye the speed of which is beyond acertain threshold; a blurring of the image used for eye tracking; ablurring of the image used for eye tracking in combination with a fastmovement; a cumulated difference between subsequent tracking imageslying beyond a certain threshold; a tracking error which is detectedbased on one or more images being considered statistically as outliers;a sequence of images with a highs speed retinal movement; an imagesequence pattern representing a saccade; an image sequence patternrepresenting a blink.
 16. The apparatus of claim 14, wherein saiddetermination whether an anomaly is detected is performed in realtime,and if an anomaly is detected said diagnostic or surgical device is putinto a HOLD mode.
 17. The apparatus of claim 14, wherein said anomalyduring eye tracking is based on retrospectively evaluating images whichhave been obtained during eye tracking to identify anomalies, saidapparatus further comprising: a module for reperforming the operation ofsaid surgical or diagnostic device on those locations for whichanomalies have been detected.
 18. The apparatus of claim 17, whereinsaid retrospective evaluation comprises: comparing image sequencesobtained during operation of said surgical or diagnostic device with eyemovement patterns representing common eye movements to detect saidanomalies.